DLG Section: Experimentation design: Part 2

Audience & sample size

Define exactly which audience segment you will be testing with.

Key Considerations:

• Who will participate in the experiment? (e.g., new users, returning customers, high-intent visitors)
• How will the audience be selected? (e.g., organic traffic, paid campaigns, segmentation by geography or behaviour)
• Are there any exclusion criteria? (e.g., users who have already interacted with a similar feature)

Determining Sample Size

The sample size should be large enough to ensure statistical significance. A small sample may lead to inconclusive results, while a large sample may require unnecessary resources.

Steps to Determine Sample Size:

1. Identify baseline conversion rates.
2. Use a sample size calculator (e.g., Optimizely, SurveyMonkey) to determine the required sample size.

Implementation and A/A Test

Implementation Plan

Successful experiment implementation requires a structured approach. Each experiment should have:

• A clear experiment setup documented.
• Necessary engineering and design changes planned.
• Tracking and analytics correctly implemented.

Key Implementation Steps:

1. Set up the test within an experimentation platform (e.g., Google Optimize, Optimizely, VWO).
2. Ensure proper event tracking via analytics tools (e.g., Google Analytics, Mixpanel, Amplitude).
3. Test the setup before launching to detect any inconsistencies.
4. Deploy the experiment gradually if necessary (e.g., rolling out to 10% of traffic first).

A/A Test for Validation

An A/A test is conducted before launching an A/B test to validate the experiment setup. In an A/A test, run two identical versions of an experiment to baseline against each other. If the results are not the same, there is something wrong with the experiment setup. This could be right from the instrumentation to the way the users have been distributed.